Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CreatePipeline instead of PipelineSpec for resources.Pipeline struct #2287

Merged
merged 5 commits into from
Feb 7, 2025

Conversation

andrewnester
Copy link
Contributor

Changes

CreatePipeline is a more complete structure (superset of PipelineSpec one) which enables support of additional fields such as run_as and allow_duplicate_names in DABs configuration. Note: these fields are subject to support in TF in order to correctly work.

Tests

Existing tests pass + no fields are removed from JSON schema

@denik
Copy link
Contributor

denik commented Feb 4, 2025

Is it possible to use new API stubbing functionality (example: https://github.com/databricks/cli/blob/main/acceptance/workspace/jobs/create/test.toml#L6) to add a couple of acceptance tests:

  • one that uses run_as attribute (the one we want)
  • one that uses dry_run attribute (the one that we don't want to support)

Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it would be nice to drop dry_run and allow_duplicate_names from the schema, but we have examples of other resources with properties we don't intend to support so we can deal with that separately.

Just so that we can omit these properties later, could you make sure that they're not forwarded to TF as part of tfdyn? That way we can remove them later (when we also take a closer look at properties included for other resources) and not regress anything.

@andrewnester
Copy link
Contributor Author

Is it possible to use new API stubbing functionality

@denik Once support for bundle deploy in acceptance test lands + TF change for run_as is done and available in CLI, it indeed would be useful. Before that we could only add the one for validation config but this is less useful at the moment

Copy link
Contributor

@shreyas-goenka shreyas-goenka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me other than the issue of losing documentation.

bundle/internal/schema/annotations_openapi_overrides.yml Outdated Show resolved Hide resolved
// will be exposed which is expected and test will need to be updated.
RunAs: &pipelines.RunAs{
UserName: "foo@bar.com",
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this PR is a no-op until the TF bump happens?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it just uses a new struct and make sure we don't have schema changes for now

bundle/deploy/terraform/tfdyn/rename_keys.go Outdated Show resolved Hide resolved
bundle/internal/schema/annotations.yml Show resolved Hide resolved
Copy link
Contributor

@ilyakuz-db ilyakuz-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@andrewnester andrewnester added this pull request to the merge queue Feb 7, 2025
Merged via the queue into main with commit 5aa8923 Feb 7, 2025
9 checks passed
@andrewnester andrewnester deleted the pipeline-spec-change branch February 7, 2025 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants